CLAIMS 



What is claimed is: 

1. A computer-implemented method for providing voice-based and text- 
based functionality to a mobile wireless client computing device from multiple 
remote servers in a coordinated manner, the client device having a current user 
and supporting a data mode for exchanging text-based information with a remote 
data server system and supporting a voice mode for exchanging voice-based 
information with a remote voice server system, the method comprising: 

under control of the mobile wireless client computing device, 

establishing a data connection with the data server using the data 
mode of the client device, and receiving multiple text-based documents from the 
data server over the established data connection; 

receiving an indication from the cun-ent user of a selection of one of 
multiple voice-based actions for a selected one of the text-based documents, 

automatically sending to the data server over the established data 
connection an indication of the selected document and of the selected action; and 
without further indication by the client device of the selected action 
or the selected document, automatically establishing a voice connection with the 
voice server using the voice mode of the client device and receiving voice-based 
information from the voice server over the established voice connection to effect 
the selected voice-based action for the selected text-based document; 

under control of the remote data server system and after receiving the 
indication from the client device of the selected text-based document and of the 
selected voice-based action, automatically sending context information for the 
current user to the voice server, the context information including an indication of 
the selected text-based document and of the selected voice-based action; and 

under control of the remote voice server system and after receiving the 
context infomnation for the current user from the data server and after the 
establishment of the voice connection with the client device. 
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identifying that the current user is using the client device; 

automatically associating the received context information for the 
current user with the established voice connection with the client device; and 

providing the selected voice-based action for the selected text-based 
document to the current user based at least in part on the associated context 
information, the providing including automatically sending the voice-based 
information to the client device over the established voice connection, 
so that a user of a mobile wireless client computing device can automatically 
receive voice-based functionality for a selected text-based document based at 
least in part on context information for the user that is automatically provided by a 
data server to a distinct voice server. 

[c2] 2. The method of claim 1 including, under the control of the remote 

voice server system and after the providing of the selected voice-based action to 
the current user of the client device: 

sending multiple groups of voice-based information to the client device over 
the established voice connection; 

receiving an indication from the client device of a selected voice-based 
information group and of a selected text-based action for the selected voice-based 
information group; and 

automatically sending context information for the current user to the data 
server, the context Information including an indication of the selected voice-based 
information group and of the selected text-based action, so that the data server 
can automatically provide the selected text-based action for the selected voice- 
based information group to the client device over the established data connection. 

[c3] 3. The method of claim 1 wherein the automatic establishing by the 

client device of the voice connection with the voice server using the voice mode of 
the client device includes ending the established data connection with the data 
server using the data mode of the client device, and including, after the providing 
of the selected voice-based action for the selected text-based document to the 
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current user of the client device, automatically re-establishing the data connection 
between the data server and the client device using the data mode in such a 
manner as to return the current user to a previous state of interaction with the data 
server based on the context information for the current user. 

[c4] 4. The method of claim 1 including, after the providing of the selected 

voice-based action for the selected text-based document to the current user of the 
client device, repeatedly exchanging updated context information for the current 
user between the voice server and the data server so that interactions by the data 
server with the current user of the client device over the established data 
connection can be maintained in a synchronous manner with interactions by the 
voice server with the current user of the client device over the established voice 
connection. 

[c5] 5. The method of claim 1 wherein the data server is an email server 

and the voice server is a unified messaging system, wherein the data connection 
is based on TCP/IP protocol and the voice connection is a digital audio protocol 
for cellular telephones, wherein the selected text-based document is an email 
message to the current user from a sender, and wherein the selected voice-based 
action is to record a voice reply message to the email message for forwarding to 
the sender. 

[c6] 6. The method of claim 1 wherein the data server is an email server 

and the voice server is a voice messaging system, wherein the selected text- 
based document is an email message having a header and contents, and wherein 
the selected voice-based action is to audibly read to the current user at least the 
contents of the email message. 

[c7] 7. The method of claim 1 wherein the selected text-based document 

includes one or more records from a database, and wherein the selected voice- 
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based action is to record a voice annotation to be stored in the database in a 
manner associated witli the records. 

[c8] 8. The method of claim 1 wherein the context information for the 

current user sent by the data server to the voice server further includes an 
indication of the current user and of a manner of the establishing of the voice 
connection between the client device and the voice server. 

[c9] 9. The method of claim 1 wherein the automatic establishing by the 

client device of the voice connection with the voice server includes, under the 
control of the client device, receiving an instruction from the data server to initiate 
the establishment of the voice connection with the voice server and automatically 
contacting the voice server to initiate the voice connection in response to the 
received instruction. 

[ciO] 10. The method of claim 1 including: 

under control of a second client device of the current user that is distinct 
from the mobile wireless client computing device, 

establishing a second data connection with the data server and 
receiving information from the data server over the established second data 
connection that is non-text and non-voice, the established second data connection 
occurring concurrently with the established data connection with the mobile 
wireless client computing device; and 

sending to the data server over the established second data 
connection an indication of a second voice-based action selected by the current 
user for the information received by the second client device; and 

under the control of the data server and after receiving the indication of the 
second voice-based action, sending to the voice server a second set of context 
information for the current user that includes an indication of the second voice- 
based action and of the second client device and of the information received by 
the second client device, so that the voice server can provide to the current user 
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based on the second set of context information the second voice-based action 
over a voice connection established with the mobile wireless client computing 
device. 

[cii] 11. The method of claim 1 including, under the control of the data 

server: 

receiving a request from a second user of the client device for a voice- 
based action related to indicated non-voice information; 

updating context information for the second user to reflect the received 
request, the updated context information including an indication of the requested 
voice-based action and of the indicated non-voice information; and 

sending the updated context information for the second user to the voice 
server, so that the voice server can provide the requested voice-based action 
based on the updated context information over a voice connection established 
with the client computing device. 

[ci2] 12. A computer-implemented method for providing functionality from a 

voice server system and a data server system in a coordinated manner, the voice 
server system able to exchange voice-based information with multiple client 
devices using a voice transmission protocol designed for transmitting voice-based 
data, the data server system able to exchange text-based information with multiple 
client devices using a distinct data transmission protocol, the method comprising, 
under control of the data server system: 

providing multiple predefined text-based groups of information to a client 
device, the providing part of a data session with the client device using the data 
transmission protocol; 

receiving from the client device an indication of one of the text-based 
groups of information and an indication of one of multiple voice-based types of 
functionality related to the indicated text-based group of information, the receiving 
part of the data session; and 
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automatically providing to the voice server system context information that 
includes an indication of the indicated text-based group of information and of the 
indicated voice-based type of functionality, 

so that the voice server system can automatically provide the indicated voice- 
based type of functionality for the indicated text-based group of information as part 
of a voice session with the client device using the voice transmission protocol. 

[ci3] 13. The method of claim 12 including, under control of the voice server: 

receiving the provided context information from the data server system; and 
providing to the client device the indicated voice-based type of functionality 
related to the indicated text-based group of information, the providing part of the 
voice session with the client device using the voice transmission protocol and 
based on the provided context information. 

[ci4] 14. The method of claim 13 wherein the providing of the indicated voice- 

based type of functionality to the client device under the control of the voice server 
includes initiating the voice session with the client device. 

[ci5] 15. The method of claim 12 including, under control of the client device, 

receiving the indicated voice-based type of functionality for the indicated text- 
based group of information as part of the voice session with the voice server 
system using the voice transmission protocol. 

[ci6] 16. The method of claim 12 including, under control of the client device 

and after providing the indication to the data server system of the one text-based 
group of information and of the one voice-based type of functionality, initiating the 
voice session with the voice server system in order to receive the provided voice- 
based type of functionality from the voice server system, the initiating based at 
least in part on an instruction received from the data server system in response to 
the providing of the indication to the data server system. 
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[ci7] 17. The method of claim 12 wherein the voice session between the 

client device and the voice server system is established before an indication is 
received from a user of the client device of the one text-based group of 
information and the one voice-based type of functionality, and wherein that 
previously established voice session is used for the providing of the indicated 
voice-based type of functionality to the client device from the voice server system. 

[ci8] 18. The method of claim 12 wherein the context information provided to 

the voice server system further includes indications of one or more of the client 
device, a user of the client device, preference information associated with the user 
of the client device, preference information associated with the client device, an 
application executing on the data server system with which the indicated one text- 
based group of information is associated, and information related to establishing 
of the voice session between the client device and the voice server system. 

[ci9] 19. The method of claim 12 wherein the data session is based on a 

connection between the data server system and the client device using a data 
mode of the client device, and wherein the voice session is based on a connection 
between the voice server system and the client device using a voice mode of the 
client device. 

[c20] 20. The method of claim 19 wherein providing to the client device the 

indicated voice-based type of functionality for the indicated text-based group of 
information includes automatically switching from the data mode of the client 
device to the voice mode of the client device for interactions between the client 
device and the voice server system that are part of the voice session. 

[c2i] 21. The method of claim 20 including, after the providing to the client 

device of the indicated voice-based type of functionality for the indicated text- 
based group of information, automatically switching from the voice mode of the 
client device back to the data mode of the client device for further interactions 



between the client device and the data server system that are part of the data 
session. 

[c22] 22. The method of claim 20 including, after the providing to the client 

device of the indicated voice-based type of functionality for the indicated text- 
based group of information, maintaining the voice session with the voice server 
system to allow further interactions between the client device and the voice server 
system that are part of the voice session. 

[c23] 23. The method of claim 19 wherein the client device has one or more 

additional modes other than the voice and data modes, and including establishing 
communications between the client device and one or more remote server 
systems using one or more of the additional modes. 

[c24] 24. The method of claim 12 wherein the indicated text-based group of 

information is an email message. 

[c25] 25. The method of claim 12 wherein the indicated text-based group of 

information is one or more records from a database. 

[c26] 26. The method of claim 12 wherein the indicated voice-based type of 

functionality provided by the voice server system is storing voice-based 
information received from a user of the client device. 

[c27] 27. The method of claim 12 wherein the indicated voice-based type of 

functionality provided by the voice server system is providing voice-based 
information to the client device. 

[c28] 28. The method of claim 12 including: 

providing one or more additional groups of information to the client device 
as part of the data session with the data server system using the data 
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transmission protocol, the additional groups of information representing 
information other than text or audio; and 

after receiving from the client device an indication of one of the additional 
groups of information and an indication of one of multiple voice-based types of 
functionality related to the indicated one group of information, providing to the 
voice server system context information that includes an indication of the indicated 
one group of information and of the indicated voice-based type of functionality 

[c29] 29. The method of claim 28 wherein the additional groups of information 

each represent one of an image, a video clip, or a graphic. 

[c30] 30. The method of claim 12 including repeatedly providing context 

information to the voice server system from the data server system to enable 
ongoing coordination of interactions between the client device and the voice 
server system over the voice session with interactions between the client device 
and the data server system over the data session. 

[c3i] 31. The method of claim 12 wherein the indicated voice-based type of 

functionality for the indicated text-based group of information includes providing 
voice-based information to the client device that is related to the indicated text- 
based group of information. 

[c32] 32. A computer-readable medium whose contents cause a server 

computing device of a first type to provide functionality in a coordinated manner 
from one or more other servers, communications between the server computing 
device of the first type and one or more clients occurring in a distinct manner than 
communications between the other servers and one or more clients, the providing 
of the functionality under control of the server computing device by performing a 
method comprising: 

receiving from a client an indication of a group of information indicated by a 
user and an indication of a selected functionality related to the indicated group of 

35 



information, the receiving of the information based on a first manner of 
communication between the client and the server computing device, the selected 
functionality available from one or more of the other servers when a client 
communicates with at least one of those other servers in a second manner that is 
associated with the selected functionality and that is distinct from the first manner 
of communication; and 

automatically providing context information to at least one of the one or 
more other servers that includes an indication of the indicated group of information 
and of the selected functionality, 

so that the one or more other servers can automatically provide the selected 
functionality for the indicated group of information when a client of the user and 
those other servers communicate in the distinct second manner of communication 
associated with the selected functionality. 

[c33] 33. The computer-readable medium of claim 32 wherein the server 

computing device is a voice server and the other servers are data servers, 
wherein the first manner of communication between the client and the server 
computing device is based on a voice session using a voice mode of the client, 
wherein the second manner of communication between the client and the one or 
more other servers is based on one or more data sessions using a data mode of 
the client, wherein the indicated group of information is voice-based information 
selected by the user, and wherein the selected functionality is non-voice-based 
functionality. 

[c34] 34. The computer-readable medium of claim 32 wherein the server 

computing device is a data server and the other servers are voice servers, 
wherein the first manner of communication between the client and the server 
computing device is based on a data session using a data mode of the client, 
wherein the second manner of communication between the client and the one or 
more other servers is based on one or more voice sessions using a voice mode of 
the client, wherein the indicated group of information is non-voice-based 
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information selected by the user, and wherein the selected functionality is voice- 
based functionality. 

[c35] 35. The computer-readable medium of claim 32 wherein the indicated 

group of information is an email message. 

[c36] 36. The computer-readable medium of claim 32 wherein the first and 

second manners of communication by the client device use two distinct 
communication modes of the client device. 

[c37] 37. The computer-readable medium of claim 32 wherein the computer- 

readable medium is a memory of a computing device. 

[c38] 38. The computer-readable medium of claim 32 wherein the computer- 

readable medium is a data transmission medium transmitting a generated data 
signal containing the contents. 

[c39] 39. The computer-readable medium of claim 32 wherein the contents 

are instructions that when executed cause the computing device to perform the 
method. 

[c40] 40. The computer-readable medium of claim 32 wherein the indicated 

group of information is of a first type that is selected by the user and/or provided 
by the user for storage, and wherein the related selected functionality available 
from one or more of the other servers via the distinct manner of communication 
includes storing and/or providing a distinct group of information of a second type 
that is associated with the indicated group of information. 
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[c4i] 41. A computing system for providing functionality in a coordinated 

manner from multiple servers of distinct types, comprising: 

a voice server that is capable of exchanging voice-based information with 
client devices using a voice-based transmission protocol; 

a data server that is capable of exchanging non-voice-based information 
with client devices using a data-based transmission protocol; and 

a context provider component that is capable of at least one of, 

A) after the data server has received an indication from a client 
device using the data-based transmission protocol of a selected type of 
functionality that is available from the voice server using the voice-based 
transmission protocol, providing information to the voice server indicating the 
selected type of functionality and at least one of the client device and a user of the 
client device, the information provided to the voice server enabling the voice 
server to automatically provide the selected type of functionality using the voice- 
based transmission protocol without the client device and/or the user having to 
further indicate the selected type of functionality to the voice server; and 

B) after the voice server has received an indication from a client 
device using the voice-based transmission protocol of a selected type of 
functionality that is available from the data server using the data-based 
transmission protocol, providing information to the data server indicating the 
selected type of functionality and at least one of the client device and a user of the 
client device, the information provided to the data server enabling the data server 
to automatically provide the selected type of functionality using the data-based 
transmission protocol without the client device or the user having to further 
indicate the selected type of functionality to the data server. 

[c42] 42. The computing device of claim 41 wherein the context provider 

component is executing in memory of the computing device. 

[c43] 43. A computing system for providing functionality in a coordinated 

manner from a voice server and a data server, the voice server capable of 
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exchanging voice-based information with clients using a voice transmission 
protocol, the data server capable of exchanging other types of information with 
clients using a distinct data transmission protocol, comprising: 

means for, after one of the servers has provided information to a client 
using the transmission protocol for that server, receiving from the client using that 
transmission protocol an indication of a selected type of functionality related to the 
information, the selected functionality type available from the other server using 
the transmission protocol for that other server; and 

means for automatically providing context information to that other server 
that includes an indication of the selected type of functionality and of the related 
information and of an identifier associated with the client, 

so that the other server can automatically provide to the client the selected type of 
functionality for the related information using the transmission protocol for that 
other server. 

[c44] 44. A method for providing information to client devices from a voice 

server system and a data server system in a coordinated manner, the voice server 
capable of exchanging voice-based information with client devices using a voice 
mode, the data server capable of exchanging text-based information with client 
devices using a distinct data mode, the method comprising: 
under control of the data server, repeatedly, 

exchanging text-based information as part of a data session with a 
client device of a user using the data mode; 

providing information to the voice server that indicates at least some 
of the exchanged text-based information; 

receiving information from the voice server that indicates voice- 
based information exchanged by the voice server with a client device of the user 
using the voice mode as part of a simultaneous voice session; and 

selecting additional text-based information to be provided as part of 
the data session with the client device using the data mode, the selecting based at 
least in part on the received information from the voice server; and 
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under control of the voice server, repeatedly, 

exchanging voice-based information as part of the voice session with 
the client device of the user using the voice mode; 

providing information to the data sen/er that indicates at least some 
of the exchanged voice-based information; 

receiving information from the data server that indicates text-based 
information exchanged by the data server with the client device using the data 
mode as part of the data session; and 

selecting additional voice-based information to be provided as part of 
the voice session with the client device using the voice mode, the selecting based 
at least in part on the received information from the data sen/er. 
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